PyDABs codegen: autogenerate core resource wiring - #6397
Merged
Conversation
Collaborator
Integration test reportCommit: cfeb0a7
Top 3 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-autogen-wiring
branch
from
August 27, 2026 21:22
7f66409 to
1acc08f
Compare
Sankalp-Mittal
changed the base branch from
sankalp-mittal/pydabs-mark-generated-files
to
sankalp-mittal/pydabs-catalogs
August 27, 2026 21:22
Sankalp-Mittal
marked this pull request as ready for review
August 27, 2026 22:51
The PyDABs codegen writes databricks/bundles/<ns>/_models/*.py and each namespace __init__.py, but those files carried no generated-file marker. - Prepend a "# Code generated by pydabs-codegen. DO NOT EDIT." header in both writer paths (_write_code, _write_exports) and regenerate. - Add a nested python/databricks/bundles/.gitattributes marking the generated _models trees and namespace __init__.py files as linguist-generated (core/ and resources/ stay hand-written). Addresses review comments on #6335. Co-authored-by: Isaac <no-reply@databricks.com>
Match all namespace __init__.py files and unset the hand-written core one, instead of listing each generated namespace. New resource namespaces are then marked automatically without touching .gitattributes. Co-authored-by: Isaac <no-reply@databricks.com>
Wire the catalog resource into the core package: add_catalog, the catalogs collection property, catalog_mutator, and the _ResourceType registration, mirroring the other resources. The catalog model was already generated (resources.Catalog is in RESOURCE_NAMESPACE) but never wired, so it was not user-addable. Add a catalog case to test_resources.py. Catalog is supported only on the direct deployment engine (there is no terraform converter for it); this matches YAML-defined catalogs. Co-authored-by: Isaac <no-reply@databricks.com>
Validate that a YAML-defined catalog and a Python-added catalog coexist, serialize, and are mutated by catalog_mutator. Restricted to the direct deployment engine, since catalogs have no terraform converter. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Rewrite Resources.add_resource and Resources.add_resources to iterate _ResourceType.all() instead of enumerating each resource type by hand, so they no longer need per-resource edits when a resource is added. Behavior-preserving. Co-authored-by: Isaac <no-reply@databricks.com>
Replace the per-type Resources._jobs/_pipelines/... attributes with a single self._resources dict keyed by plural name, populated by iterating _ResourceType.all(). __init__ no longer needs a per-resource line; the add_X methods and properties read/write self._resources[plural]. Behavior-preserving. Co-authored-by: Isaac <no-reply@databricks.com>
Replace the per-type "for a job" / "for an alert" phrasing in the duplicate-resource-name error with a uniform "for resource '<type>'". This removes the grammatical-article special case so the wiring can be generated uniformly. Drops the now-unused TestCase.article field. Co-authored-by: Isaac <no-reply@databricks.com>
Add codegen/generated_wiring.py, which emits the per-resource wiring for databricks.bundles.core into a new core/_generated/ package: the _ResourceType registry (_resource_types.py), the Resources add_*/property methods (_resources.py, a mixin), and the *_mutator decorators (_resource_mutators.py). main.py calls it after the model-generation loop. The core package __init__ is now generated too (static exports plus the generated mutator exports); mark core/_generated/ generated in .gitattributes and wipe it before regenerating. The hand-written wiring in _resource_type.py / _resources.py / _resource_mutator.py is removed in the next commit. Co-authored-by: Isaac <no-reply@databricks.com>
Resources now inherits the generated _GeneratedResources mixin instead of defining per-resource add_*/property methods; _ResourceType.all() returns the generated tuple; and _resource_mutator.py keeps only the ResourceMutator dataclass, with the decorators now generated. Tests import the mutators from the public databricks.bundles.core instead of the internal module. Behavior-preserving: the generated wiring reproduces what was hand-written. Co-authored-by: Isaac <no-reply@databricks.com>
Address review feedback that the f-string-based generator was hard to reason about. Each wired resource now gets its own _generated/<plural>.py rendered from wiring_resource.py.tmpl (a string.Template with $-placeholders, so the generated shape reads like real Python and there is no brace-escaping). The generated _generated/__init__.py collects the per-resource mixins into _GeneratedResources, exposes _all_resource_types(), and re-exports the mutators. Per-resource files stay small as more resources are onboarded instead of growing single aggregate modules. Behavior unchanged: same generated API, tests green, regeneration idempotent. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-autogen-wiring
branch
from
August 28, 2026 12:35
1def02a to
d385c7f
Compare
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-autogen-wiring
branch
from
August 28, 2026 14:26
d385c7f to
6feddc6
Compare
…-autogen-wiring # Conflicts: # python/databricks/bundles/.gitattributes # python/databricks/bundles/core/__init__.py # python/databricks/bundles/core/_resource_mutator.py # python/databricks/bundles/core/_resource_type.py # python/databricks/bundles/core/_resources.py # python/databricks_tests/core/test_resources.py
Sankalp-Mittal
force-pushed
the
sankalp-mittal/pydabs-autogen-wiring
branch
from
August 28, 2026 14:31
6feddc6 to
cfeb0a7
Compare
Sankalp-Mittal
changed the base branch from
main
to
sankalp-mittal/pydabs-public-api-guard
August 31, 2026 09:34
…-guard' into sankalp-mittal/pydabs-autogen-wiring
_bases() filtered out `object` but not underscore-prefixed private bases, so a generated private base (e.g. _GeneratedResources introduced by the wiring refactor) would surface in the golden as a false positive even though the public contract is intact. Mirror _members()'s underscore filter. No-op for the current golden on this branch, where Resources has no private base. Co-authored-by: Isaac <no-reply@databricks.com>
…-guard' into sankalp-mittal/pydabs-autogen-wiring
_bases() filtered out `object` but not underscore-prefixed private bases, so a generated private base (e.g. _GeneratedResources introduced by the wiring refactor) would surface in the golden as a false positive even though the public contract is intact. Mirror _members()'s underscore filter. No-op for the current golden on this branch, where Resources has no private base. Co-authored-by: Isaac <no-reply@databricks.com>
The core public-API snapshot doesn't exercise the CLI end to end, so it belongs with the other pure-Python pydabs tests rather than in acceptance/. Move it to databricks_tests/core/test_public_api.py, comparing against a committed golden (regenerate with UPDATE_SNAPSHOTS=1). Gate on Python >= 3.11 (via skipif, and a sys.version_info guard so pyright at 3.10 is happy) instead of pinning an exact interpreter, since typing.get_overloads needs 3.11+ and the output is identical on 3.11/3.12/3.13. Removes the acceptance/bundle/python/public-api test. Co-authored-by: Isaac <no-reply@databricks.com>
The whitespace linter (task checks -> ws) strips a trailing blank line and then git diff --exit-code fails; in acceptance/ the golden was in the ws skip-list, but as a committed .txt under python/ it is checked. Emit a single trailing newline. Co-authored-by: Isaac <no-reply@databricks.com>
…-guard' into sankalp-mittal/pydabs-autogen-wiring
Adds a scoped .gitattributes so GitHub renders the codegen wiring
templates as Python (the ${...} placeholders won't resolve, but the
rest highlights). Addresses reviewer feedback on #6397.
Co-authored-by: Isaac <no-reply@databricks.com>
The per-resource files already came from wiring_resource.py.tmpl, but _generated/__init__.py and core/__init__.py were still built from inline f-strings in generated_wiring.py. Move those into generated_init.py.tmpl and core_init.py.tmpl so all generated output is rendered from templates (and highlights as Python). Generated output is unchanged. Addresses reviewer feedback on #6397. Co-authored-by: Isaac <no-reply@databricks.com>
janniklasrose
approved these changes
Sep 3, 2026
Base automatically changed from
sankalp-mittal/pydabs-public-api-guard
to
main
September 3, 2026 10:53
hejcman-enverus
pushed a commit
to hejcman-enverus/databricks_cli
that referenced
this pull request
Sep 3, 2026
…ks#6439) ## Summary A regression guard has been added to the public facing API's on PyDABs, this is done to test that auto generating the code does not change any public API's. ## Why The upcoming PyDABs codegen/wiring refactor (databricks#6397) converts the hand-written `databricks.bundles.core` wiring — `Resources`, the `*_mutator` functions, the `_ResourceType` registry, `__all__` — into generated code. Nothing today guards the *typed public surface* customers import and type-check against, so that refactor (or a future one) could silently drop a type hint, move a `*` keyword marker, rename a method, or change the export set. This lands a golden snapshot of that surface **on main first**, freezing the known-good pre-refactor API. databricks#6397 then merges main in and must reproduce the identical golden — that passing test is the proof the refactor preserved the public API. `generate-check` cannot do this: after the refactor regenerates the wiring it trivially passes ("checked-in == regenerated"); only a golden captured here, before the refactor, can show the *new generated* wiring equals the *old hand-written* surface. ## What An acceptance test at `acceptance/bundle/python/public-api/` whose `script` runs a checked-in `dump_public_api.py` under `uv`, snapshotting `databricks.bundles.core`'s `__all__`, every `Resources.add_*` signature and property, the `*_mutator` overloads, `Variable`/`VariableOr*`, the support types, and the `_ResourceType.all()` registry. ## Notable design decisions - **Types render by public short name** (`Variable[str]`, not `databricks.bundles.core._variable.Variable`). The refactor moves internal `_`-prefixed modules; a golden keyed on internal paths would false-fail even when the public API is unchanged. Public imports go through the `core.__init__` re-exports, which is exactly what the golden pins. - **Signatures are reconstructed** from `inspect.Signature` so `/`, `*`, `*args`, `**kwargs` markers render explicitly and stably. - **`--python 3.11` is pinned** so the golden is reproducible independent of the repo-wide `UV_PYTHON` minimum, and `typing.get_overloads` (3.11+) is available. Output verified identical on 3.11 and 3.12. - Runtime error strings are deliberately excluded — that's behavior, not the typed API. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generate the
databricks.bundles.coreper-resource wiring instead of hand-writing it, so adding a PyDABs resource needs no manualcore/edits — onlyRESOURCE_NAMESPACE. Stacked on #6408.Changes
codegen/codegen/wiring_resource.py.tmpl, astring.Template) into its owncore/_generated/<resource_plural>.py— theadd_<x>()+ collection-property mixin, the<x>_mutatordecorator, and a lazy_resource_type(). A generatedcore/_generated/__init__.pycollects them into_GeneratedResources(mixed intoResources),_all_resource_types(), and the mutator re-exports.core/__init__.pyis generated too.Resources.add_resource/add_resources/__init__are now data-driven off_ResourceType.all()(singleself._resourcesdict), so they don't grow per resource.for resource '<type>'wording, dropping the a/an special case.Behavior-preserving apart from that wording. Generalizing the hardcoded
jobs.ForEachTaskrecursion handling is left as a follow-up.Tests
Unit +
codegen_testspass;./task pydabs-codegenis idempotent (no_models/diff); pyright andruffclean.This pull request and its description were written by Isaac.